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1.  SUMMARY: 


This  research  project  investigated  techniques  to  develop  a  High  Performance  Computing 
(HPC)  grid  infrastructure  to  operate  as  an  interactive  research  and  development  tool. 
Current  HPC  grid  architectures  are  designed  for  batch  applications,  where  users  submit 
their  job  requests,  and  then  wait  for  notification  of  job  completion.  In  the  batch 
environment,  the  user  lacks  direct  control  over  the  execution  of  their  application.  While 
this  may  be  acceptable  for  running  certain  very  large  computationally  intense  batch  jobs 
on  large  mainframe  HPCs,  it  is  not  acceptable  for  use  with  jobs  that  require  a  near  real¬ 
time  response  and  an  interactive  environment.  To  meet  this  need  for  accessing  and 
processing  data  interactively  in  near  real-time,  the  Air  Force  Research  Laboratory/ 
Information  Directorate  developed  an  environment  that  stresses  ’near  real-time’  user 
interaction  with  the  application.  This  involved  evaluating  the  various  developing 
protocols  for  interactive  grid  computing,  using  the  Globus  Toolkit,  and  then  selecting  the 
one  with  the  most  growth  potential.  The  grid  architecture  was  evaluated  by  assembling 
and  demonstrating  an  in-house  interactive  demonstration  grid  using  in-house  cluster 
assets  and  existing  code,  to  verify  proper  operation  on  a  small  scale.  This  project 
provided  a  framework  for  longer  term  efforts  to  investigate  and  improve  interactive 
scalability  and  perfonnance,  taking  into  account  the  needs  of  the  Joint  Battlespace 
Infosphere,  high  perfonnance  computing,  and  logistics. 


2.  INTRODUCTION: 


This  research  project  investigated  techniques  to  develop  a  High  Performance  Computing 
(HPC)  grid  infrastructure  to  operate  as  an  interactive  research  and  development  tool.  [5] 
Current  HPC  grid  architectures  are  designed  for  batch  applications,  where  users  submit 
their  job  requests,  and  then  wait  for  notification  of  job  completion.  In  the  batch 
environment,  the  user  lacks  direct  control  over  the  execution  of  their  application.  While 
this  may  be  acceptable  for  running  certain  very  large  computationally  intense  batch  jobs 
on  large  mainframe  HPCs,  it  is  not  acceptable  for  use  with  jobs  that  require  a  near  real¬ 
time  response  and  an  interactive  environment.  With  the  current  proliferation  of  HPC 
clusters  and  the  existence  of  the  internet,  which  allows  for  remote  access  to  compute 
resources,  the  demand  for  rapid  response  times  is  escalating  and  achievable.  To  meet  this 
need  for  accessing  and  processing  data  interactively  in  near  real-time,  AFRL/Information 
Directorate  (AFRL/IF)  developed  an  environment  that  stresses  'near  real-time’  user 
interaction  with  the  application. 

The  approach  for  developing  such  an  HPC  grid  capability  involved  evaluating  the  various 
developing  protocols  for  interactive  grid  computing,  and  then  selecting  the  one  with  the 
most  growth  potential.  The  Globus  Toolkit  was  selected  for  the  grid  development 
environment.  This  toolkit  is  rapidly  becoming  the  de-facto  standard  for  grid  research. 

The  grid  architecture  was  evaluated  by  assembling  and  demonstrating  an  in-house 
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interactive  demonstration  grid  using  in-house  cluster  assets  and  existing  code,  to  verify 
proper  operation  on  a  small  scale. 


Potential  users  of  this  product  include  developers  of  experimental  systems  as  well  as 
users  who  want  to  showcase  their  codes  by  making  them  available  through  the 
Hyperspectral  Image  Exploitation  (HIE)  Framework.  [2-10,  12-16].  The  primary 
emphasis  was  on  creating  a  working  interactive  grid  for  use  with  the  HIE  framework. 

This  project  provided  a  framework  for  longer  term  efforts  to  investigate  and  improve 
interactive  scalability  and  performance,  taking  into  account  the  needs  of  the  Joint 
Battlespace  Infosphere  (JBI),  HPC,  logistics,  etc.  It  is  envisioned  that  user  interaction 
will  be  web  based,  to  assure  extensive  portability. 

To  validate  successful  operation  of  the  interactive  grid  implementation,  the  Hyperspectral 
Image  Exploitation  (HIE)  framework  software,  was  used  [10].  Truth  data  already  exists 
for  this  code,  thus  simplifying  the  final  data  analysis.  This  architecture  also  has  the 
potential  to  be  applied  to  streamline  the  management  of  Air  Force  logistics,  which 
currently  have  about  20  thousand  people  at  each  of  the  three  Air  Logistics  Centers,  all  of 
which  use  legacy  software. 

The  major  payoffs  potentially  include:  Setting  the  groundwork  for  grid-based  application 
of  the  HIE  framework,  improved  real-time  feedback  for  decision  analysis  tools,  reduced 
latency  for  publication  of  subscription  based  data  queries,  and  decreased  turn-around 
time  for  decision  support  tools,  wargaming,  and  modeling  and  simulation  tools.  An 
interactive  grid  gives  the  applications  a  more  responsive  underlying  architecture  to 
leverage  future  grid  connected  hardware. 

The  limitations  for  Grid  computing  over  HPC  Centers  mostly  arise  from  security 
concerns  and  the  requirement  to  access  heterogeneous  environments.  Even  when  systems 
are  identical,  the  grid  mechanisms  that  deal  with  security  were  difficult  to  implement  and 
couldn’t  be  adequately  addressed  within  the  scope  of  this  project.  Also,  significant 
additional  problems  arise  when  hardware  and  software  heterogeneity  are  considered. 
Other  limitations  are  related  to  the  difficulty  in  assigning  processing  and  establishing 
reservations  across  groups  of  systems,  for  example  at  distinct  HPC  centers.  Considerable 
additional  work  is  necessary  to  consider  these  issues.  This  project  explored  the  basic  Grid 
capabilities,  leaving  these  additional  issues  for  future  consideration. 


3.  APPROACH: 


Three  in-house  workstations  were  configured  for  software  development.  Two  were  Linux 
based  and  one  ran  Microsoft  Windows.  Of  the  two  Linux  workstations,  one  was 
configured  as  a  “grid”  server.  Communication  between  the  server  and  the  development 
workstation  was  established  over  a  secure  channel  using  Virtual  Network  Computing 
(VNC)  based  on  Tight  VNC  and  using  a  Secure  Shell  (ssh)  connection  from  a  Linux 
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workstation  running  Red  Hat  Linux.  Tight  VNC  supplied  the  ssh  and  graphical  (X-based) 
connection  to  the  server.  There  was  an  issue  with  resource  intensive  connections  using 
Tight  VNC. 

To  address  this  issue,  the  hardware  was  upgraded  from  the  original  ‘grid’  server,  named 
“mintho”  with  a  1.2  GHz  CPU  and  512  MBs  of  random  access  memory  (RAM)  to  a  new 
server  named  “styx”  with  2  central  processing  units  (CPUs)  operating  at  2+  GHz  and  4 
GBs  of  RAM.  It  was  hosting  the  installation  of  the  VNC  Server  software  and  the  Globus 
Toolkit  (version  3.2).  The  various  required  ancillary  tools  were  installed,  and  tested.  [17] 
Graphical  secure  communication  (using  VNC)  between  mintho  and  a  Windows  2000 
desktop  system  and  a  Linux  desktop  system  was  demonstrated.  The  team  evaluated  the 
intricacies  of  Globus  installation  and  determined  an  effective  way  to  install  Globus 
version  3.2  on  the  server.  This  provided  access  to  all  the  necessary  and  sufficient  tools  to 
submit  and  interact  with  Grid  (-like)  job  submissions.  The  next  step  was  integration  of  a 
web-browser  interface  to  Globus,  providing  the  ability  to  launch,  interact-with,  tenninate, 
etc.  grid  applications  in  semi-real-time.  The  Air  Force  Research  Laboratory, 

Information  Directorate  (AFRL/IF),  successfully  integrated  the  Globus  Grid  toolkit  with 
the  HIE  framework.  One  of  the  framework  objects  was  rewritten  as  a  grid  service  and 
presented  at  the  2005  Object-Oriented  Programming,  Systems,  Languages,  and 
Applications  (OOPSLA)  Conference  [1].  Additional  work  explored  using  Grid 
technology  and  information  management  for  command  and  control  [16]. 

The  architecture  diagram  below,  Figure  1,  depicts  the  experimental  setup  used  for  the 
project.  In  the  diagram,  the  virtual  network  computer  (VNC)  connection  provides  access 
from  a  Linux  or  Windows  2000  computer  to  the  Grid  Services  Container.  A  “secure 
shell”  (ssh)  connection  was  also  used  to  tunnel  messages  between  Linux  and  the  Globus 
“container”  (server)  system.  The  diagram  shows  a  “counter”  service  operating  within  the 
Globus  Grid  Services  Container.  It  shows  that  “stubs”,  compiled  into  client  software, 
provide  access  to  the  services  that  are  active  in  the  Grid  Services  Container.  Services  are 
accessed  via  Skeleton  interfaces  that  are  presented  by  the  Grid  Container  as  Grid 
services.  The  stub/skeleton  basis  for  the  architecture  relieves  developers  of  the 
requirement  to  implement  detailed  network  interfaces,  providing  a  significant  opportunity 
for  developers  to  implement  efficient  interfaces  easily.  The  stub  and  skeleton  are 
automatically  generated  from  a  Web  Services  Description  Language  (WSDL)  interface 
description. 
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Figure  1:  Architecture  Diagram 


Notice  that  all  references  to  the  counter  service  are  given  using  global  web  references 
that  include  the  name  of  the  system  and  a  path  that  uniquely  identifies  a  service  available 
through  a  Grid  Services  Container.  In  the  example  3  below,  The  Grid  services  container 
is  listening  on  port  8080  on  styx  and  has  a  service  available  called 
ogsa/services/guide/counter/CounterFactoryService/calc  which  can  accept  input 
parameters,  passed  by  the  client ,  to  specify  which  of  the  available  “calc”  functions 
should  be  invoked. 

The  following  examples  show  how  the  grid  process  works.  The  first  three  examples 
below  show  how  to  invoke  Grid  Services.  Examples  4  and  5  demonstrate  the  built-in 
capabilities  for  Grid  Management  and  Administration.  Services  can  be  started  and 
stopped  using  the  administrative  interfaces  to  interact  with  the  Grid  Services  Container. 
Sample  computer  outputs  from  the  installation  and  test  are  shown  below.  The  four 
examples  demonstrate  initializing  the  grid  services  environment,  creating  and  accessing  a 
local  service,  accessing  a  remote  service  on  styx  (from  astro),  and  using  the  Globus 
Service  Browser  to  verily  services  that  are  active  in  the  Globus  Grid  Services  Container. 
AFRL/IF  also  used  the  Globus  notification  services  to  set  up  a  client  that  is  notified 
whenever  the  counter  changes  (not  shown  in  the  examples  below). 

Notice  that  all  references  to  the  counter  service  are  given  using  global  web  references 
that  include  the  name  of  the  system  and  a  path  that  uniquely  identifies  a  service  available 
through  a  Grid  Services  Container.  In  the  example  3  below,  the  Grid  services  container  is 
listening  on  port  8080  on  styx  and  has  a  service  available  called 
ogsa/services/guide/counter/CounterFactoryService/calc  which  can  accept  input 
parameters,  passed  by  the  client,  to  specify  which  of  the  available  “calc”  functions  should 
be  invoked. 
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EXAMPLE  1:  Initializing  Variables 


Example  1  shows  how  to  initialize  environment  variables  that  are  needed  by  the  grid 
services  container  and  also  by  the  service  browser  tool  shown  in  the  screengrabs  in 
example  4. 


Set  up  the  Grid  Services  environment 


cd  /usr/local/gt3 
.  ~scott/.bash_profile 
.  setenv.sh 

globus-start-container 

globus-service-browser 


//  Setup  Java/Grid  Environment 

//  Start  Java  Container 
//  Start  Gui  to  show  the  state  of  services 
//  Screengrab  before  starting  Counter  services 
//  and  after  starting  them  are  shown  below. 


EXAMPLE  2:  Creating  Counter  Service 

This  example  demonstrates  creating  the  counter  service  and  client  application  access  for 
the  counter  service. 

Create  the  counter  service: 

The  following  two  lines  contain  the  command  to  create  the  Globus  counter  service: 
ogsi-create-service 

http://localhost:8080/ogsa/services/guide/counter/CounterFactorvService/calc 

Use  the  counter  service,  adding  10 

The  following  lines  contain  the  command  to  use  the  Globus  counter  service: 
java  org. globus. ogsa. guide. impl. CounterClient 

http://localhost:8080/ogsa/services/guide/counter/CounterFactoryService/calc 

add  10 

»  output »  Counter  add:  10 


EXAMPLE  3:  Running  Counter  Service 
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We  got  the  counter  service  running  on  styx  by  doing  "ant  deployGuide"  in  the  gt3 
directory.  We  added  that  to  the  instructions  that  are  given  below  that  show  how  to  run, 
not  deploy,  the  counter  service. 

The  lines  below  show  that  we  were  able  to  use  the  counter  service  on  styx  from  a  client 
counter  application  on  astro.  Notice  that  the  user  command  prompt  indicates  that  the 
client  is  running  on  astro  but  the  specified  service,  provided  as  an  argument  to  the 
CounterClient  client  application,  is  running  on  styx.oc.rl.af.mil. 

Get  the  counter  value  from  styx: 

The  following  command  gets  the  counter  value  from  styx: 

[scott@astro  gt3]$  java  org. globus. ogsa.guide.impl. CounterClient  http://styx.oc 
.rl.af.mil:8080/ogsa/services/guide/counter/CounterFactoryService/calc  get 

Countervalue:  10 

Note  that  the  counter  value  changes  (subtraction  performed  by  a  different  client 
program) 

[scott@astro  gt3]$  java  org. globus. ogsa.guide.impl. CounterClient  http://styx.oc 
.rl.af.mil:8080/ogsa/services/guide/counter/CounterFactoryService/calc  get 

Counter  value:  5 

[scott@astro  gt3]$ 


EXAMPLE  4:  Using  Globus  Browser 

To  demonstrate  the  built-in  capabilities  for  Grid  Management  and  Administration  with 
the  Globus  Service  Browser,  Example  4  shows  how  to  use  the  Globus  Service  Browser  to 
verify  services  that  are  active  in  the  Globus  Grid  Services  Container.  Services  can  be 
started  and  stopped  using  the  administrative  interfaces  to  interact  with  the  Grid  Services 
Container.  To  illustrate  use  of  the  browser,  Figures  2,  3,  and  4  show  screenshots  of  the 
Globus  Services  Browser.  Each  of  these  figures  illustrates  an  aspect  of  the  functionality 
available  through  the  Globus  Service  Browser. 


Figure  2  shows  that  the  counter  service,  including  notification  services,  is  inactive  (notice 
CounterFactoryService,  WSDLCounterFactoryService, 
ServiceDataCounterFactoryService,  NotificationCounterFactoryService,  etc.): 
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//  ServiceBrowserInactiveCounter.jpg  shows 
//  the  Inactive  counter  services 


-  H  X 


New  Window 


Refresh  □  Show  dynamic  gui 


|http: //astro,  oc.  rl.af.  mil:  8080/ogsa/seryices/core/registrv/ContainerRegistryService 


Services  WSDL  Service  Data 


Grid  Service 
Namespace: 


Name: 


Timeout: 


XPath  Expression: 


XPath  Namespace  Mappings: 


Query 


Subscribe 


Unsubscribe 


Timeout(seconds): 


Destroy 


Service  Group  Entry  Inspection 
|f  Table  ServiceGroup  Entries 


Name 

Handle 

State 

guide/counter/CounterProviderFactorySen/ice 

http://128.132.2.206:8080/ogsa/services/guide/counter/CounterProviderFactor\5... 

INACTIVE 

guide/cou  nter/ServiceDataAnnotation  C  o  u  nterFactoiyService 
guide/counter/ComplexCounterFactoryService 

http://128.132.2.206:8080/ogsa/services/guide/counter/ServiceDataAnnotationCo... 
http:// 12  8. 132.2.206:8080/ogsa/services/guide/counter/ComplexCounterFactory5... 

INACTIVE 

INACTIVE 

guide/counter/NotificationCounterFactoryService 

guide/counter/ServiceDataCounterFactoryService 

guide/counter/WSDLCounterFactoryService 

http:// 12  8. 132.2.206:8080/ogsa/services/guide/counter/NotificationCounterFactor... 
http://128.132.2.206:8080/ogsa/services/guide/counter/ServiceDataCounterFacto... 
http:// 12  8. 132.2.206:8080/ogsa/services/guide/counter/WSDLCounterFactory5ervice 

INACTIVE 

INACTIVE 

INACTIVE 

guide/counter/CounterFactoryService 

core/admin/AdminService 

http://128.132.2.206:8080/ogsa/services/guide/counter/CounterFactory5ervice 

http://128.132.2.206:8080/ogsa/services/core/adrnin/AdminService 

INACTIVE 

INACTIVE 

- 

n.  Antn  nnrfatp  1  Rpfrp^h  .1 


Figure  2:  Counter  Service  Inactive 
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Figure  3  shows  that  the  counter  service,  CounterFactoryService,  is  now  active  and  one 
instance  of  the  CounterFactoryService  “calc”  object  has  been  created. 

//  ServiceBrowserActiveCounter.jpg 
//  Shows  that  counter  is  active 
//  The  CounterFactoryService  and  one  instance 
//  of  the  Counter  Service  are  active 


Figure  3:  Counter  Factory  Service 


Figure  4  shows  that  the  NotificationSubscriptionFactoryServices  are  active: 


//  ServiceBrowserOtherServices.jpg 
//  Shows  that  Flandle  Resolver, 

//  Notification  Subscription  Service  and 
//  Generic  Persistent  Grid  Service  are 
//  active 


x' 

Forward 

New  Window 

Close 

Refresh 

□  Show  dynamic  gui 

|http: //astro,  oc.rl.af.  mil:  8080/ogsa/services/core/registry/ContainerRegistr\5ervice 

Go 

Services  WSDL  Service  Data 

Hi  - 

- II; 

- 

Query 

Subscribe 

Unsubscribe 

Grid  Service 
Namespace: 


Name: 


XPath  Expression: 


XPath  Namespace  Mappings: 


Timeout(Seconds): 


Service  Group  Entry  Inspection 
|i  Table  ServiceGroup  Entries 


Set 


Get 


Destroy 


Name 


Handle 


State 


Generic  Persistent  Grid  Service 


htt  p :  /  / 1 2  3 . 1 3  2 . 2 . 2  0  6 : 8  0  8  0  /  o  g  s  a/  s  e  rvi  c  e  s  /  o  g  s  i  /  N  ot  if  i  c  at  i  o  ns  u  b  s  c  ri  pt  i  o  n  Fact  o  ry5  e  rvi  c  e 


Notification  Subscription  Service 


http:// 12  8. 132.2.206:8080/ogsa/services/ogsi/NotificationsubscriptionFactor\Servi.. 


ACTIVE 


Handle  Resolver 


http ;  /  / 12  8. 132.2.2  0  6 : 8  0  8  0  /  o  g  s  a  /  s  e  rvi  c  e  s  /  o  g  s  i  /  H  an  d  I  e  R  e  s  o  I ve  rs  e  rvi  c  e 


base/gram/ResourcelnformationProviderService 


http:// 12  8. 132.2 .206: 8080/ogsa/services/base/gram/ResourcelnformationProvide.. 


INACTIVE 


b  ase/index/ln  dexService 


http: // 12  8. 132 .2.2  06: 8080/ogsa/services/base/index/lndexService 


b  ase/se  rvi  c  e  g  r  o  u  p /Servi  c  e  Gro  u  p  F  acto  ry 


base/servic  e  g  roup/ServiceGroupService 


http:// 12  8. 132.2.206: 
http://128.132. 2. 206: 


8080/ogsa/services/base/servicegroup/ServiceGroupFactorY 
8080/ogsa/services/base/servic  egroup/ServiceGroupService 


INACTIVE 


INACTIVE 


base/streaming/FileStreamFactoryfactoryService 


http://128.132.2.206:8080/ogsa/services/base/streaming/FileStreamFactorV:actor.. 


INACTIVE 


f~  Antn  unriatp 


Figure  4:  Notification  Subscription  Factory  Services 

These  services  together  constitute  the  necessary  steps  to  initiate  and  run  an  interactive 
grid  job,  by  initializing  the  grid  services  environment,  creating  and  accessing  a  local 
service,  accessing  a  remote  service  on  the  Grid  Services  Container,  and  using  the  Globus 
Service  Browser  to  verify  services  active  in  the  Globus  Grid  Services  Container.  This  is 
a  significant  accomplishment  that  moves  grid  computing  towards  interactive  computing. 


4.  THE  HIE  FRAMEWORK  GRID  INTERFACE: 
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The  HIE  FrameWork  implements  a  general  purpose  Web  interface  for  access  to  remote 
applications,  usually  running  on  HPCs.  The  Web  interface  is  driven  by  objects  that 
implement  interfaces  to  specific  codes,  allowing  the  Web  interface  to  collect  application- 
specific  inputs  and  deliver  them  to  remote  applications  appropriately.  This  section 
describes  the  steps  that  use  grid  services  to  implement  FrameWork  application  interface 
objects.  The  application  interface  object  is  used  by  the  FrameWork  server  to  determine 
the  variables  that  have  to  be  supplied  through  the  Web  interface.  Figure  5  shows  the 
FrameWork  flow.  In  the  diagram  the  Code  Dependent  User  Services  represent  the  code 
interface  object.  Notice  that  the  code  interface  object  provides  information  to  the  User 
Interface  Services,  or  FrameWork  server,  that  describes  the  inputs  to  be  collected  from 
the  user.  The  User  Inputs  are  then  used  to  request  execution  service  as  shown  in  Figure  5. 


User  Interface 
Services 


Code  Dependent 
User  Services 


Code  Execution 
ftgent  Services 


FrameWork  Flow 


Figure  5:  Frame  Work  Flow 

The  original  FrameWork  implementation  used  a  Corba  interface  to  connect  the 
FrameWork  server  with  the  code  interface  object.  For  this  project,  we  added  a  Grid 
interface  to  allow  developers  to  implement  code  interface  objects  using  the  Globus  Grid 
software  or  the  Corba  software.  For  both  Globus  Grid  and  Corba,  the  code  to  implement 
the  interface  and  establish  the  connection  between  the  FrameWork  server  and  the  code 
interface  object  can  be  generated  automatically  from  an  interface  description  that 
specifies  remote  functions  (implemented  in  the  code  interface  object)  and  parameters  that 
are  needed  to  call  the  functions.  The  FrameWork  server  can  then  access  the  remote 
function  without  regard  for  network  issues,  like  locating  the  remote  object  and  opening  a 
connection,  by  simply  calling  the  function.  For  the  FrameWork,  the  code  interface  object 
is  implemented  as  a  service  that  waits  for  service  requests  generated  by  the  Globus  Grid 
or  Corba  software  that  is  executed  as  a  result  of  a  request  by  the  FrameWork  server  to 
execute  an  interface  function. 


In  the  examples  below,  we  focus  on  the  FwGetFunc  function  that  is  implemented  in  the 
code  interface  object.  After  the  user  selects  one  of  the  codes  to  execute,  the  code 
interface  object  must  return  a  list  of  functions  available  for  that  code.  Each  of  the 
functions  may  require  a  different  set  of  input  parameters.  The  code  interface  object  for 
the  selected  code  must  return  the  information  needed  by  the  FrameWork  server  so  that  it 
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can  present  a  list  of  the  available  functions  to  the  user,  through  the  Web  browser 
interface.  Once  a  particular  function  is  selected,  the  FrameWork  server  uses  the  same 
code  interface  object  that  also  must  implement  the  FwGetParams  function,  so  that  the 
FrameWork  server  can  request  the  specific  list  of  parameters  that  are  needed  by  the  code 
from  the  user,  through  the  Web  browser  interface. 

5.  GENERATING  STUBS  AND  SKELETONS: 

For  both  Globus  and  Corba,  a  standard  interface  description  is  used  to  automatically 
generate  the  interface  functions  needed  by  the  client  (called  a  stub)  and  the  server  (called 
a  skeleton).  Generally,  all  of  the  service  code  needs  to  be  built  into  the  server,  by 
implementing  the  code  that  is  executed  when  the  service  interface  (skeleton)  is  invoked. 
The  term  skeleton  seems  appropriate  since  the  interface  code  is  just  a  container  in  which 
the  service  must  be  implemented.  The  stub,  by  contrast  is  just  the  client’s  interface  to 
make  the  call  and  pass  the  parameters  to  the  service  implementation.  In  Corba,  the 
interface  is  specified  using  a  standard  called  Interface  Definition  Language  (IDL).  The 
Grid  uses  an  Ex(tensible)  M(arkup)  L(anguage)  XML  XSLT  specification.  Extensible 
stylesheet  language  transfonnation  (XSLT)  is  a  language  for  transforming  XML 
documents  into  other  XML  documents.  XSLT  is  designed  for  use  as  part  of  XSL,  which 
is  a  stylesheet  language  for  XML.  The  example  XML  XSLT  code  for  our  Grid  interface 
to  the  fwGetFunc  interface  for  the  code  interface  object  is  shown  in  Figure  6. 


Generate  Client  Stubs  and  Service 

Skeleton  <xsd:element  name="fwGetFunc"> 
<xsd:complexType> 

<xsd:sequence> 

<xsd:element  name="Opcode"  type="xsd:string"/> 

<xsd:etement  name=”Mesg"  type=“xsd:string"/> 

<xsd:element  name="Fname"  ty pe="xsd : string" '/> 

</xsd:sequence> 

</xsd:complexType> 

</xsd:element> 


<xsd:element  name= "fwGetFuncResponse  "> 
<xsd:complexType> 

<xsd:sequence> 

<xsd:element  name="Retval"  type=“xsd:string"/> 
</xsd:sequence> 

</xsd:complexType> 

</xsd:eiement> 


Figure  6:  Specifying  the  Interface 

6.  A  GLOBUS  FUNCTION  STUB: 


The  stub  is  used  by  the  client  to  call  a  function  that  is  actually  implemented  as  a  service 
in  the  grid  services  container.  The  code  shown  in  Figure  7  was  generated  by  a  Globus 
utility  program  for  the  fwGetFunc  interface  to  the  code  interface  object  that  is  described 
above. 
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fwGetFunc. Opcode  =  argv[2]; 
fwGetFunc.Mesg  =  argv[3]; 
fwGetFunc. Fname  =  argv[4]; 

/*  create  blog  resource  with  createBlogTopic 
operation  */ 

result  =  Fw_fwGetFunc( 
fw_handle, 
argv[1], 

&fwGetFunc, 

&fwG  etFuncResponse, 
&create_fault_type, 

&fault); 


/*  destroy  response  from  fwGetFunc  */ 
fwGetFuncResponseType_destroy(fwGetFuncResponse); 

printf("Returning  from  fwGetFunc\n"); 

/*  destroy  client  handle  */ 
FwService_client_destroy(fw_handle); 

rc  =  globus_module_deactivate(FWSERVICE_MODULE); 
if(rc  !=  0) 

{ 

globus_fatal("FwService  deactivate  failed"); 

} 

exit(0); 

} 


Figure  7:  Globus  Client  Stub 

7.  A  GLOBUS  FUNCTION  SKELETON: 


The  skeleton  function  is  compiled  and  installed  in  the  C  Web  Services  Container.  Users 
can  add  backend  code  to  perform  desired  services.  The  part  of  the  skeleton  shown  in 
Figure  8  initializes  debugging  and  can  be  used  to  perform  custom  initialization  of  the 
service  if  necessary.  The  Fw  fwGetFunc  imp  function  implements  the  service  that  is 
compiled  into  the  C  Web  Services  Container  and  sends  results  back  to  the  client  program, 
through  the  client  stub  interface,  by  copying  it  into  an  “xsd  string  Retval”  variable 
generated  from  the  interface  description. 
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FwServiceJnit 

Fw_SetTerminationTime_ 

Fw_Destroy_init 

Fw_GetCurrentMessage_ 

Fw_Subscribe_init 

Fw_fwGetResults_init 

Fw_fwExecFunc_init 

Fw  fwGetParams  init 


FwService_finalize 
init  Fw_SetTerminationTime_impl 
Fw_Destroy_impl 

init  Fw_GetCurrentMessage_impl 
Fw_Subscribe_impl 
Fw_fwGetResults_impl 
Fw_fwExecFunc_impl 
Fw_fwGetParams_impl 


Fw_fwGetFunc_init( 

globus_service_engine_t  engine, 

globus_soap_message_handle_t  message, 
fwGetFuncType  *  fwGetFunc) 

{ 

/*  add  function  local  variable  declarations  here  7 
globus_result_t  result  =  GLOBUS_SUCCESS; 

/*  initialize  trace  debugging  info  7 

GlobusFuncName(Fw_fwGetFunc_init); 

FwServiceDebugEnter(); 

/* 

*  If  no  configuration  or  initialization  needs  to  be  done,  this 

*  call  can  remain  empty. 

7 

FwServiceDebugExitQ; 
return  GLOBUS_SUCCESS; 

} 

globus_result_t 
Fw_fwGetFunc_impl( 
globus_service_engine_t  engine, 

globus_soap_message_handle_t  message, 
globus_service_descriptor_t  *  descriptor, 
fwGetFuncType  *  fwGetFunc, 
fwGetFuncResponseType  *  fwGetFuncResponse, 
const  char  **  fault_name, 
void  **  fault) 

{ 

/*  add  function  local  variable  declarations  here  7 
globus_result_t  result  =  GLOBUS_SUCCESS; 


13 


/*  initialize  trace  debugging  info  */ 
GlobusFuncName(Fw_fwGetFuncJmpl); 
FwServiceDebugEnter(); 

/*  This  is  where  it  all  happens.  Service  implementer  must 

*  implmenent  this  function.  Asume  that  fwGetFunc  has 

*  been  initialized  and  filled  with  request  values. 

*  fwGetFuncResponse  must  be  set  by  the  implementer. 

*/ 

/*  Use  the  error  object  construction  api  */ 

result  =  FwServiceErrorNotlmplemented("Fw_fwGetFunc"); 

FwServiceDebugExit(); 
return  result; 

} 


Figure  8;  Globus  Service  Skeleton 


8.  FUTURE  VISION  -  INTERGRID:  A  GRID  OF  GRIDS: 

This  project  has  paved  the  way  for  future  work  on  an  intergrid  or  a  grid  of  grids, 
allowing  even  greater  expansion  of  the  network  over  which  jobs  can  be  run  and  providing 
the  basis  for  an  experimental  testbed  where  new  concepts  and  techniques  in  parallel  and 
distributed  computing  can  be  explored. 

8.1  Grid  Technology  Applications: 

Grid  technology  has  matured  to  the  point  that  there  are  now  services  that  directly 
support  access  to  HPCs.  These  services  use  the  Global  Resource  Access  Manager 
protocol,  implemented  in  the  Globus  toolkit,  to  negotiate  processing  resources  and  to 
submit  jobs  through  standard  HPC  batch  access  systems.  Grid  protocol  implementations 
are  widespread,  making  them  a  defacto  standard  for  modern  distributed  computing. 

8.2  Applications  of  Grid  Protocols: 

Grid  protocols  have  a  wide  range  of  potential  applications.  Some  of  the  most 
promising  are  for  the  Teragrid,  a  large  grid-based  effort,  originally  funded  by  the  NSF  to 
connect  five  large  computing  facilities  and  other  sites.  [16]  Other  applications  include 
local  grids,  and  multiple  site  grids,  such  as  hosted  by  the  DoD  High  Performance 
Computing  Modernization  Program  (HPCMP)  or  the  Condor  Program. 

8.3  New  Computing  Models: 

This  work  leads  to  the  development  of  new  computing  models.  For  example, 
since  tasks  are  distributed  among  a  number  of  different  systems,  a  director  of  task 
distribution  is  used  to  meet  performance  and  reliability  requirements.  In  directing  task 
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distribution,  the  performance  is  monitored  to  ensure  that  at  least  one  of  the  computations 
will  finish  in  time;  and  the  reliability  is  monitored  to  make  sure  that  at  least  one 
computation  will  complete  correctly.  The  perfonnance  monitoring  required  for  this 
approach  could  leverage  a  system  such  as  the  AFRL  Distributed  Interactive  High 
Performance  Computing  Testbed  (DIHT). 

A  workstation  model  was  tested  at  AFRL/IF;  with  additional  development 
currently  being  researched  at  SUNY  Institute  of  Technology,  under  the  guidance  of  Dr. 
Spetka.  With  jobs  being  distributed  over  a  large  number  of  systems,  cluster  reliability  can 
be  enhanced,  since  several  systems  redundantly  execute  the  job. 

With  the  high  cost  of  large  computer  systems,  and  the  ability  to  run  these  systems 
24/7,  investing  in  large  grids  of  systems  as  national  shared  resources,  to  be  used  by  many 
different  organizations,  is  a  cost  effective  way  to  make  high  performance  computing 
available  to  many  users.  Grid  computing  models  can  also  incorporate  privately  owned 
high  perfonnance  computing  grids,  with  charges  for  use  based  on  cycles  used  or  other 
metrics. 

As  data  and  code  are  shared  between  several  different  sites  on  a  grid,  guarding  the 
confidentiality  and  security  of  code  and  data,  to  only  enable  access  to  authorized 
personnel,  is  important. 

8.4  Security  Considerations: 

They  are  currently  used  on  the  Teragrid.  To  be  part  of  the  TeraGrid  national 
cyberinfrastructure  resource  providers  are  responsible  for 
(http://www.teragrid.org/basics/): 

*  support  for  TeraGrid  data  movement  services 

*  participation  in  security  coordination  and  implementation  of  policies 

*  assisting  in  problem  resolution  for  issues  related  to  local  resources 

*  support  for  the  Coordinated  TeraGrid  Software  and  Services  specification 

*  participation  in  verification  and  validation  processes  and 

*  participation  in  the  allocations  and  accounting  processes 

Other  application  communities  include  the  HPCMP  community  and  the  Condor-G 
system.  The  Globus  Grid  security  mechanisms  can  support  Kerberos  and  are  likely  to  be 
approved  for  use  for  HPCMP  access.  HPCMP  centers  are  developing  support  for  public 
key  infrastructure  (PKI)  and  we  expect  that  it  will  be  approved  eventually  to  open  up 
implementing  Grid  services.  Condor-G  is  another  grid-based  system  that  uses  both 
Condor  and  Globus  to  enable  users  to  access  multiple  systems  as  if  they  are  just  one.  [1] 

8.5  Recommendations: 

Further  work  can  be  performed  by  experimenting  with  a  heterogeneous  model. 
Future  funding  may  be  sought  to  experiment  with  HPCMP  centers.  AFRL  is  in  a  unique 
position  to  leverage  distributed  and  parallel  computing  expertise,  including  the  DIHT  and 
the  HIE  to  evolve  an  integrated  parallel  and  distributed  computing  system  where  we 
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envision  that  all  future  systems  will  play  a  role.  Figure  9  shows  an  InterGrid 
Environment. 


Workstations 


Figure  9:  InterGrid  Environment 


9.  CONCLUSIONS: 

This  research  project  successfully  set  up  and  operated  an  interactive  grid  using  Globus 
Toolkit,  ran  it,  and  prepared  the  way  for  subsequent  integration  with  the  HIE  framework. 
This  is  a  significant  step  up  from  traditional  HPC  grid  architectures,  which  are  designed 
for  batch  applications,  where  the  user  lacks  direct  control  over  the  execution  of  their 
application.  Applications,  such  as  the  HIE  framework  require  a  near  real-time  response 
and  an  interactive  environment.  Lessons  learned  were  how  to  successfully  use  the  Globus 
toolkit  to  run  these  services,  including  initializing  variables,  creating  and  running  the 
counter  service,  and  using  the  Globus  browser.  Future  work  in  the  area  is  to  use  this 
capability  to  run  interactive  grid  based  work,  supporting  the  warfighter  in  areas  such  as 
the  Joint  Battlespace  Infosphere. 
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